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Background 


It  has  long  been  recognized  that  survey  administration  can  affect  indi¬ 
vidual  responses  to  various  questions  (Cronbach,  1970;  Duncan  1947;  Melton, 
1947).  Such  things  as  the  condition  of  the  testing  room,  control  of  the 
group,  and  directions  to  the  subjects  have  all  been  shown  to  influence 
scores.  The  administration  of  the  Organizational  Assessment  Package  (OAP) 
is  subject  to  many  of  these  same  biases.  It  will  be  the  purpose  of  this 
paper  to  more  fully  understand  the  effect  of  taking  the  survey  at  different 
points  in  time  in  relation  to  one's  normal  duty  schedule. 

Although  Leadership  and  Management  Development  Center  Regulation  10-4, 
the  Organizational  Assessment  Package  Survey  Program  (1979,  p.  3-1)  mandates 
that  “data  collection  must  be  done  as  consistently  as  possible  from  time  to 
time  and  from  base  to  base  to  insure  comparability  of  the  data",  many  tech¬ 
nical  and  logistical  problems  prevent  absolute  compliance.  Indeed,  most 
project  officers  at  the  client's  base  fail  to  or  are  unable  to  schedule  par¬ 
ticipants  during  their  normal  duty  hours  as  th^y  are  directed  to  do  in  the 
Managment  Consultation  Project  Officer's  Guide  (1980).  Reasons  given  for 
scheduling  an  individual  or  work  group  at  a  time  other  than  duty  hours  are 
typically  related  to  mission  requirements,  and  appear  to  be  unavoidable 
obstacles.  Hence,  it  is  important  to  understand  the  effect  of  taking  the 
survey  during  nonduty  hours.  Since  alert  subjects  are  more  likely  to  give 
their  best,  what  is  the  effect  of  getting  up  an  hour  and  a  half  early  to 
take  a  survey  prior  to  beginning  a  shift?  Since  fatigue  has  been  shown  to 
affect  motivation  to  take  a  survey,  what  is  the  effect  of  having  worked  a 
normal  shift  only  to  face  an  hour-long  survey?  If  one  is  temporarily  upset 
at  the  prospect  of  staying  after  duty  hours  to  take  a  survey,  what  is  the 


effect  on  one's  responses  on  the  OAP?  Indeed,  what  if  one  were  required  to 
report  to  work  on  a  "training  day"  to  take  the  OAP?  Information  regarding 
these  questions  was  the  motivation  for  the  following  analyses. 

Method 

Investigation  1 

Subjects.  Based  on  the  nature  of  the  job.  Air  Force  security  police 
often  are  unavailable  during  their  normal  duty  hours  for  long  enough  periods 
of  time  to  take  the  OAP.  At  one  base,  an  effort  was  made  to  identify  those 
who  were  required  to  take  the  survey  (a)  before  duty  hours,  (b)  during  nor¬ 
mal  duty  hours,  (c)  after  duty  hours,  (d)  on  a  training  day.  The  number  of 
subjects  for  the  entire  sample  are  listed  in  Table  1. 

Procedure.  A  series  of  oneway  analyses  of  variance  were  performed  to 
ascertain  differences  that  might  exist  among  the  groups  across  the  800 
series  variables  as  identified  in  the  OAP  Output  (Appendix). 

The  data  within  the  OAP  data  base  contains  survey  information  collected 
since  January  1979  by  the  Air  Force  Leadership  and  Management  Development 
Center  (LMDC).  The  109  question  survey  was  designed  by  the  Air  Force  Human 
Resources  Laboratory  to  aid  LMDC  in  its  mission  to:  (a)  provide  management 
consulting  services  to  Air  Force  commanders  upon  request,  (b)  to  provide 
leadership  and  management  training,  and  (c)  to  conduct  research  on  Air  Force 
systemic  issues  with  information  within  the  accumulated  data  base  (Hendrix 
and  Halverson,  1979). 


Table  1 

Number  of  Subjects  by  Treatment  Condition 
Investigation  1 

Before  Duty  257 

During  Duty  81 

After  Duty  213 

Training  Day 


135 


Administration  of  the  survey  is  the  first  step  in  the  consultation  pro¬ 
cess.  The  survey  is  given  to  a  stratified  random  sample  of  the  organization 
to  which  LMDC  has  been  invited.  The  results  of  the  survey  are  an  important 
feature  in  the  assessment  of  the  organization.  The  results  are  handled  in  a 
confidential  manner  between  LMDC  and  the  client.  Feedback  is  then  provided 
to  the  commander  and  every  supervisor  within  the  organization. 

When  specific  problems  are  encountered,  a  consultant  and  supervisor 
develop  a  management  action  plan  designed  to  reduce  the  problem  at  that 
level  of  the  organization.  Within  6  months,  the  consulting  team  returns  to 
readminister  the  survey  instrument  as  a  means  to  assess  the  impact  of  the 
consulting  process. 

The  data  from  each  consulting  effort  are  stored  in  an  increasing  data 
base  for  research  purposes.  These  data  are  aggregated  by  work  group  codes 
developed  for  this  instrument.  The  data  may  be  recalled  by  personnel  cate¬ 
gory,  age,  sex,  AFSC,  pay  grade,  time  in  service,  and  educational  level. 

The  remainder  of  the  items  are  combined  to  form  24  factors. 

The  oneway  analyses  of  variance  were  followed  up  by  a  Student-Newman- 
Keuls  multiple  range  test  to  further  analyze  differences.  Finally,  the  two 
most  comparable  groups  were  compared  across  all  individual  items  to  deter¬ 
mine  if  any  differences  existed.  This  anlysis  was  performed  using  a  series 
of  t-Tests. 

Investigation  2 

Subjects.  The  initial  investigation  provided  some  suspicion  in  the  com¬ 
parability  of  the  groups.  Indeed,  a  follow-up  breakdown  indicated  that  the 
initial  group  that  was  surveyed  during  duty  hours  was  more  senior  in  terms 
of  time  in  service  (Table  2).  In  this  investigation,  only  those  with  less 


Table  2 


Percentages 

of  First  Term  Airmen  Within  Groups 

Investigation  1 

Before  Duty 

81.8% 

During  Duty 

27.2% 

After  Duty 

71.1% 

Training  Day 

81.5% 

than  four  years  of  service  were  considered.  The  sample  now  contained  the 
numbers  reflected  in  Table  3. 

The  personnel  taking  the  survey  during  duty  hours  were  found  to  be 
evenly  distributed  among  the  major  work  centers  (staff,  law  enforcement,  and 
missile  security).  Additionally,  further  chi  square  analyses  yielded  no 
significant  differences  in  the  composition  of  law  enforcement  and  security 
personnel  in  either  the  Before  or  After  groups.  However,  significant  dif¬ 
ferences  (>£  =  94.58)  were  found  in  the  composition  of  the  training  group. 
The  following  analyses  must  be  viewed  with  the  knowledge  that  the  training 
group  has  an  inordinately  high  percentage  of  missile  security.  In  as  much 
as  the  primary  aim  of  this  stucty  is  to  first  analyze  the  differences  that 
exist  based  on  time  the  survey  was  taken  in  relation  to  duty  schedule,  it 
was  decided  not  to  lose  the  power  derived  from  the  sample  size;  therefore, 
the  author  elected  to  avoid  use  of  a  statistical  procedure  that  might 
account  for  the  inequality  within  this  one  group.  The  implications  of  any 
found  differences  will  be  discussed. 

Procedure.  The  same  procedure  as  identified  in  investigation  1  was  used 
with  the  exception  of  using  only  those  individuals  with  less  than  four  years 
service.  An  analysis  of  covariance  was  not  selected  for  a  number  of  rea¬ 
sons,  the  primary  one  being  the  unnecessary  complication  of  the  multiple 
comparisons  among  units. 

Results 

Investigation  1 

Results  of  the  oneway  analyses  of  variance  are  presented  in  Table  4. 

All  significant  results  are  reported  using  p  >  .05.  The  significant  differ¬ 
ences  found  using  the  Student-Newman-Keul s  range  test  are  shown  by  the  use 
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Table  4 

Oneway  Analyses  of  Variance 

Time  Survey  Administered  Across  Organizational  Factors 

V800  Skill  Variety  Mean 

SD 

Training  2.78 

1.28 

Before  3.04 

1.44 

After  3.06 

.  1.40 

During  4.54 

1  1.43 

F  (3,657)  =  30.48,  £  < 

001 

V801  Task  Identity 

■j 

After  3.65 

1.42 

Before  3.79 

1.39 

Training  3.90 

,  1.58 

During  4.94 

1.17  H 

F  (3,630)  =  17.20,  £  < 

001  i 

V802  Task  Significance 

I 

Training  4.46 

1.67 

Before  4.49 

1.74  H 

After  4.51 

1.69 

During  5.84 

1.22 

F  (3,668)  =  15.82,  £  < 

001  | 

V804  Job  Feedback 

j 

, 

After  3.53 

1.41 

Before  3.75 

1.48 

Training  3.80 

1.48 

During  4.97 

I  1.16 

F  (3,660)  =  20.90,  £  < 

001  i 

#  1 

V805  Work  Support 

During  3.92 

1.10 

Before  4.23 

1.11 

After  4.25 

1.17  -l 

Training  4.62 

1.16  H 

F  (3,629)  =  6.61,  £  <  .001 

V806  Need  for  Enrichment 

a 

.  j 

After  4.70 

Training  4.71 

t 

Before  4.79 

"1 

During  5.69 

1 

F  (3,659)  =  8.22,  £  <  .001 

8 

► 

1 


Table  4  Continued. 


Job  Motivation  Index 


Mean 

SD 

After 

43.98 

37.03 

Before 

45.45 

36.14 

Trai ni ng 

49.19 

37.96 

Duri ng 

F  (3,555)  =  59.42 

113.96 
,  £  <  .001 

1  64.92 

Job  Performance  Goals 

Before 

4.14 

1.00 

After 

4.15 

1.16 

Trai ning 

4.24 

1.05 

During 

F  (3,635)  =  6.61, 

4.72 

£  <  .001 

1  .97 

Pride 

Before 

3.45 

1.77 

After 

3.49 

1.83 

Traini ng 

3.52 

1.86 

During 

5.11 

1  1.75 

F  (3,661)  =  19.55,  £  <  .001 

Task  Characteristics 

After  3.74 

Training  3.78 

Before  3.82 

During  5.08 

F  (3,599)  =  30.19,  £  <  .001 

Task  Autonomy 


Aft  er 

2.46  1 

1.13 

Before 

2.52  1 

1.10 

Training 

2.86 

1,  1.14 

During 

4.39 

|  1.36 

F  (3,645)  =  62.18, 

£  < 

.001 

Work  Repetition 

Before 

5.38 

1.25 

Trai ni ng 

5.46 

1.42 

After 

5.60 

1.45 

During 

F  (3,669)  =  1.41, 

NS 

5.68 

1.37 

Advancement-Recognition 

Before 

3.61 

1.16 

Training 

3.72 

1.10 

After 

3.74 

1.22 

During 

4.72 

I  1.17 

F  (3,657)  =  18.60, 

£  < 

.001 
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Table  4 

Continued. 

V818  Management-Supervision 

Mean 

SD 

Before 

4.53  1 

1.52 

After 

4.80  1 

1.59 

Training 

5.10 

1.31 

During 

5.46 

1.53 

F  (3,628)  =  8.83,  £  < 

.001 

V819  Supervisory  Communications 

Cl imate 

Before 

4.01) 

1.60 

After 

4.38 

1.75 

Training 

4.62 

Ij  1.29 

During 

4.99 

1  1.69 

F  (3,639)  =  8.78,  2  < 

.001 

V820 

Organizational  Communcations  Climate 

Before 

3.56 

1.30 

After 

3.60 

1.41 

Training 

4.03 

1  1.28 

During 

4.85 

1  1.33 

F  (3,632)  =  21.56,  £ 

<  .001 

V821 

Perceived  Productivity 

After 

4.591 

1.37 

Before 

4.631 

1.38 

Training 

4.95 

I  1.22 

During 

5.69 

|  1.17 

F  (3,635)  =  15.37,  £ 

<  .001 

V822 

Job  Related  Satisfaction 

Before 

3.74 

1.25 

After 

3.88 

1.39 

Training 

4.02 

1.27 

During 

5.05 

1  1.37 

F  (3,539)  =  19.51,  £ 

<  .001 

V823 

Job  Related  Training 

Before 

3.70 

1.52 

After 

3.77 

1.52 

Training 

4.05 

1.56 

|  1.87 

During 

4.51 

F  (3,651)  =  5.96,  £  < 

.001 

10 


Table  4  Concluded 


V824  General  Organizational  Climate 


After  3.54 
Before  3.57 
Training  3.81 
During  4.96 


F  (3,636)  =  22.01,  £  <  .001 


1.45 

1.37 

1.40 

1.45 


NS  =  Differences  not  significant 


of  a  bar  technique  to  indicate  differences  among  groups.  For  example,  the 
solid  line  for  V814,  Work  Repetition  indicates  there  are  no  significant  dif¬ 
ferences  among  the  groups.  However,  the  use  of  2  bars  for  V800,  Skill  Vari¬ 
ety  indicates  that  the  Training,  Before,  and  After  subsets  are  significantly 
different  from  the  During  group.  The  means  are  provided.  One  general  trend 
emerges.  The  group  that  took  the  survey  during  duty  hours  generally  scores 
in  a  more  positive  direction. 

A  second  set  of  analyses  that  was  conducted  between  the  Before  and  After 
duty  groups.  The  results  of  the  significant  differences  are  displayed  in 
Table  5.  It  appears  that  those  individuals  that  took  the  survey  before  work 
were  less  satisfied  with  supervisory  communications  and  assistance.  These 
groups  appeared  to  be  basically  matched  in  terms  of  type  of  job,  age  and 
grade.  It  is  possible  that  having  to  report  to  work  early  as  opposed  to 
staying  a  little  late  "cost"  the  supervisor  some  "points"  in  the  minds  of 
the  subordinates. 

Investigation  2 

Results  of  the  oneway  analyses  of  variance  are  presented  in  Table  6. 

All  significant  results  are  reported  using  p  >  .05.  As  in  the  first  inves¬ 
tigation,  a  Student-Newman-Keuls  follow-up  test  was  used.  The  same  display 
format  indicates  significant  differences.  In  this  investigation  only  first 
term  airmen  were  selected.  Numerous  differences  were  still  found  among 
these  closely  matched  groups.  For  example,  those  that  took  the  survey  on  a 
"training  day"  rated  Management-Supervision,  Supervisory  Communications  Cli¬ 
mate,  Organizational  Communications  Climate  and  Work  Support  significantly 
better  than  did  one  or  more  other  groups.  While  it  may  be  argued  that  the 
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V428  r 

My  supervisor  explains  how  Before  3.90  1.80  \ 

the  job  contributes  to  After  4.33  1.94  t  =  -2.49  (df  =  462) 

overall  mission 


V433 


^  My  supervisor  lets' me  know  Before  4.20  1.99 

when  I  am  doing  a  good  job _ After _ 4.60 _ 2.08  t  =  -2.09  (df  =  458) 


>s 

» 

b 

R  *  . 

b 

[ 

V435 

My  supervisor  always  helps 
me  improve  my  performance 

Before 

After 

4.05 

4.49 

1.80 

1.91 

t  =  -2.56  (df  =  460) 

r 

V437 

My  performance  has  improved 

Before 

3.92 

1.94 

■ 

due  to  feedback  received 

After 

4.35 

2.01 

t  =  -2.31  (df  =  458) 

from  my  supervisor. 


a 


V439 

When  I  need  technical  advice 
I  usually  go  to  my  supervisor 


Before 

After 


4.17 

4.34 


V442 

My  supervisor  frequently 
gives  me  feedback  on  how 
well  I  am  doing  my  job 


Before 

After 


3.88 

4.37 


V445 

My  supervisor  fully  explains 
procedures  to  each  group 
member 


Before 

After 


4.33 

4.74 


1 . 93 

2I00  t  =  -1.99  (df  =  456) 


1.94 


2.06  t  =  -2.57  (df  =  460)  * 


1.76 


1.90  t  =  -2.39  (df  =  459) 


* 


Table  6 

Oneway  Analysis  of  Variance 

Time  Survey  Administered  Across  Organizational  Factors 
First  Term  Airmen 


V800 

Skill 

Variety 

Me  a 

Training 

2.7 

After 

2.8 

Before 

2.9 

During 

3.6 

F  (3,473)  =  2.84,  £  < 

.05 

V801 

Task 

Identity 

After 

3.5 

Before 

3.7 

Training 

3.8 

During 

4.3 

F  (3,448)  =  2.33,  NS 

V802 

Task 

Significance 

After 

4.3 

Training 

4.3 

Before 

4.4 

During 

5.3 

F  (3,477)  =  2.45,  £  < 

.05 

V804 

Job  Feedback 

After 

3.4 

Before 

3.6 

Training 

3.7 

During 

4.7 

F  (3,472)  =  5.43,  £  < 

.01 

V805 

Work 

Support 

During 

3.9 

Before 

4.2 

After 

4.2 

Training 

4.7 

F  (3,444)  =  5.53,  £  < 

.01 

V806 

Need 

for  Enrichment 

After 

4.4 

Before 

4.6 

Training 

4.7 

During 

5.3 

F  (3,472)  =  2.21,  NS 

Table  6  Continued 


V807 

Job 

Motivation  Index 

Mean 

SD 

After 

41.50 

36,48 

Before 

42.09 

34.84 

Trai ning 

47.67 

,  34.76 

During 

80.33 

1  47.39 

F  (3,392)  =  7.90, 

£  < 

.01 

V810 

Job 

Performance  Goal s 

After 

4.03 

1.17 

Before 

4.13 

.99 

Traini ng 

4.24 

.99 

Duri ng 

4.36 

1.31 

F  (3,452)  =  1.10, 

NS 

V811 

Pride 

After 

3.26 

1.81 

Before 

3.32 

1.77 

Trai ni ng 

3.50 

1.82 

During 

4.18 

2.12 

F  (3,473)  =  1.87, 

NS 

V812 

Task 

Characteristics 

After 

3.59 

1.16 

Training 

3.73 

1.10 

Before 

3.75 

1.16 

During 

4.53 

1  1.04 

F  (3,425)  =  4.21, 

£  < 

.01 

V813 

Task 

Autonomy 

Before 

2.41 1 

1.06 

After 

2.42 ' 

1.12 

Trai ning 

2.84 

!.  1.04 

During 

3.53 

1  1.19 

F  (3,464)  =  10.51 

»  £ 

<  .01 

V814 

Work 

Repetition 

Before 

5.41 

1.43 

Training 

5.56 

1.54 

After 

5.66 

1.44 

During 

6.26 

1.01 

F  (3,479)  =  2.60, 

NS 

V81 7 

Advancement-Recognit ion 

Before 

3.51 

1.14 

After 

3.62 

1.23 

Training 

3.65 

1.06 

During 

4.20 

1.48 

F  (3,480)  =  .56, 

NS 
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Table  6 

Continued 

V818 

Ma  na geme  nt  -  Su  pe  r  v  i  s  i  on 

Mean 

Before 

4.46 

During 

4.53 

After 

4.72 

Training 

5.02 

F  (3,472)  =  3.11, 

£ 

<  .05 

V819 

Supervisory  Communicati 

ons 

Cl imate 

Before 

3.96 

During 

4.06 

After 

4.30 

Training 

4.49 

F  (3,463)  =  2.79, 

£ 

<  .05 

V820 

Organizational  Communcations  Climate 

Before 

3.551 

After 

3.64  1 

Training 

4.06 

During 

4.32 

F  (3,455)  =  5.11, 

£ 

<  .05 

V821 

Perceived  Productivity 

After 

4.50 

Before 

4.57 

Training 

4.78 

During 

5.23 

F  (3,456)  =  2.46, 

NS 

V822 

Job  Related  Satisfaction 

Before 

3.67 

After 

3.82 

Training 

3.97 

During 

4.62 

F  (3,422)  =  3.96, 

£ 

<  .05 

V823 

Job  Related  Training 

During 

3.61 

Before 

3.62 

After 

3.67 

Training 

4.07 

F  (3,466)  =  2.12, 

NS 

Table  6  Concluded. 


V824  General  Organizational 

Cl  imate 

Before 

3.51 

1, 

.34 

After 

3.52 

1, 

.43 

Training 

3.77 

1  L 

.39 

During 

4.44 

1  1. 

.58 

F  (3,459)  =  3.56 

,  £  <  .05 

NS  =  Differences  not  significant 


"During"  group  is  somewhat  higher  on  many  scales  due  to  differences  in  duty 
requirements,  those  taking  the  survey  on  a  training  day  were  found  to  be 
equal  in  job  requirements  to  those  in  the  Before  duty  and  After  duty  groups. 
This  is  substantiated  by  a  review  of  the  composition  of  the  work  groups  as 
well  as  the  near  identical  scores  on  such  indexes  as  Skill  Variety,  Task  Sig¬ 
nificance,  Job  Feedback,  Job  Performance  Goals,  and  Task  Characteristics.  In 
conclusion,  it  appears  that  those  taking  the  survey  Before  duty  score  lower 
on  items  that  relate  to  their  supervisor  and  communications.  It  should  be 
noted  that  these  differences  are  often  .5  point  in  magnitude.  This  differ¬ 
ence  is  relatively  large,  and  would  often  move  a  work  group  from  an  accept¬ 
able  to  an  unacceptable  range.  Thus,  it  appears  that  those  taking  the  survey 
before  the  start  of  a  duty  shift  generally  rate  supervisory  factors  slightly 
lower,  while  those  taking  the  survey  after  duty  typically  rate  job  factors 
slightly  lower. 

The  previous  differences  noted  within  the  training  group  are  noteworthy. 
In  general,  this  group  was  found  to  contain  a  vastly  disproportionate  number 
of  missile  security  versus  law  enforcement  personnel.  Based  on  all  security 
police  data  within  the  LMOC  data  base,  one  expects  missile  security  personnel 
to  score  significantly  lower  than  law  enforcement  personnel  on  13  of  20  fac¬ 
tors.  Of  the  remaining  seven  factors,  six  should  reflect  no  differences. 

Yet,  the  training  group  (missile  security)  rated  the  supervisor  and  communi¬ 
cations  higher  than  the  Before  group,  and  Autononiy  higher  than  both  the 
Before  and  After  groups  (groups  without  differences  in  a  security-law  en¬ 
forcement  mix). 
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Discussion 


Potential  contaminants  to  the  data  of  this  selected  unit  have  been  uncov¬ 
ered.  However,  it  appears  that  the  time  an  individual  takes  the  survey  in 
relation  to  his  or  her  duty  schedule  will  impact  on  responses  on  the  OAP. 
Certain  trends  were  found  in  this  unit,  and  one  can  speculate  the  impact  of 
this  throughout  the  data  base.  As  discussed,  several  types  of  jobs  generally 
dictate  that  an  individual  will  have  to  take  the  survey  at  times  other  than 
the  duty  shift.  An  example  of  this  is  the  law  enforcement  personnel.  Per¬ 
haps  supervisory  indexes  within  these  units  are  artificially  .5  points  lower 
than  they  would  be  if  those  individuals  were  able  to  take  the  survey  during 
duty  hours. 

Surveys  given  under  these  various  conditions  do  have  value;  however,  some 
individual  work  groups'  scores  may  be  somewhat  misleading.  If  a  survey  must 
be  given  at  a  time  other  than  duty  hours,  the  only  acceptable  procedure  is  to 
document  the  fact  and  to  review  the  results  with  a  critical  eye. 

Additional  studies  in  this  area  are  recommended.  First,  the  use  of  an 
additional  sample  to  verify  these  results  is  needed.  Secondly,  a  carefully 
designed  study  should  be  developed  to  explore  some  casual  obse*vances  noted 
herein  regarding  the  Before,  After  and  training  groups:  Should  we  expect 
those  groups  to  show  the  noted  bias  consistently?  Do  those  who  must  report 
an  hour  and  a  half  early  feel  it  is  due  to  supervisory  weakness?  Are  those 
staying  late  particularly  dissatisfied  with  job  characteristics? 

At  this  time,  the  warning  is  clear.  Timing  of  survey  administration  does 
impact  survey  results. 
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NOTE:  Variable  008,  statement  11  was  added  to  the  OAP  on  19  Jan  80  and  replaced  .  1.  Yes  2.  No  3.  Not  sure 

variable  OK  which  appears  on  page  3.  Although  no  longer  used  Variable 
OK  is  still  shown  because  data  collected  from  about  25,000  samples  for 


with  a  chance  to  finish  completely  the  piece 
of  work  you  have  begun? 


FACTOR  802:  TASK  SIGNIFICANCE 


FACTOR  812:  TASK  CHARACTERISTICS 
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Value  range  will  be  from  1  to  343, 
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